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(54) Augmented reality presentation apparatus and method, and storage medium 



(57) A game state manager (201) manages the 
state of an AR game (information that pertains to ren- 
dering of each virtual object (1 02), the score of a player 
(101), the AR game round count, and the like). An ob- 
jective viewpoint video generator (202) generates a vid- 
eo of each virtual object (102) viewed from a camera 
(103). An objective viewpoint video composition unit 



(203) generates a composite video of the video of the 
virtual object (102) and an actually sensed video, and 
outputs it to a display (1 06) . A subjective viewpoint video 
generator (212) generates a video of the virtual object 
(1 02) viewed from an HMD (1 07). A subjective viewpoint 
video composition unit (21 3) generates a composite vid- 
eo of the video of the virtual object (1 02) and an actually 
sensed video, and outputs it to the HMD (1 07). 
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Description 

FIELD OF THE INVENTION 

[0001] The present invention relates to an augmented 
reality presentation apparatus and method, which su- 
perimposes a virtual object on a real space, and a stor- 
age medium. 

[0002] More specifically, the present invention relates 
to an augmented reality presentation apparatus and 
method having a function of presenting a subjective 
augmented reality view for a-player and generating a 
video of the same augmented reality space observed 
from an objective viewpoint, and a storage medium. 

BACKGROUND OF THE INVENTION 

[0003] A conventional augmented reality (AR) game 
system is constructed by an AR game apparatus for 
controlling an AR game, a head-mounted display (to be 
abbreviated as an HMD hereinafter) which the player of 
the AR game wears, and a display for presenting a sub- 
jective viewpoint video of the player to a third party other 
than the player. 

[0004] The AR game apparatus controls the AR 
game, and generates information that pertains to ren- 
dering of a virtual object at the same time. The AR game 
apparatus generates a composite video as a video ob- 
tained by compositing a video of a virtual object gener- 
ated using the information that pertains to rendering of 
the virtual object, and a video of a real space at the sub- 
jective viewpoint of the player, which is sensed by a 
camera attached to or built in the HMD. 
[0005] The composite video is displayed on the dis- 
play screen of the HMD. The player plays the AR game 
while reviewing this composite video. In addition to the 
player, the video at the subjective viewpoint of the player 
as this composite video is presented to a third party who 
watches this AR game in addition to the player by dis- 
playing that video on a display prepared for a third party 
other than the player 

[0006] Some TV programs use virtual studios in each 
of which a program stage set is built by virtual objects. 
In this virtual studio, a program video is generated by 
compositing an actually sensed video of characters of 
the program sensed by a set camera, and a video of the 
virtual studio seen at the position and posture of that 
camera. 

[0007] In the conventional AR (augmented reality) 
game system, a third party (watcher) other than the play- 
er can only watch a video generated for the player 
(mainly at the subjective viewpoint of the player). For 
this reason, the watcher cannot watch videos of the AR 
game at viewpoints other than the subjective viewpoint 
of the player, e.g., the overall view of the AR game 
viewed from the objective viewpoint, a video of the AR 
game at a desired viewpoint of the watcher, and the like 
and can neither recognize the overall status of the AR 



game nor watch the AR game from a desired viewpoint. 
[0008] Also, a program video using a virtual studio is 
generated at the objective viewpoint of a third party oth- 
er than the characters of the program, but a video 

5 viewed from the subjective viewpoint of each character 
of the program cannot be generated. 
[0009] It is, therefore, an object of the present inven- 
tion to generate a video of an AR game at the subjective 
viewpoint of the player and also generate the overall 

10 view of the AR game viewed from an objective viewpoint 
and a video of the AR game viewed from a desired view- 
point of the watcher. 

SUMMARY OF THE INVENTION 

15 

[0010] In order to achieve the above object, an aug- 
mented reality presentation apparatus described in 
claim 1 as a preferred embodiment of the present inven- 
tion comprises the following arrangement. 
20 [0011] That is, augmented reality presentation means 
for superimposing the virtual object viewed from a play- 
er's viewpoint position in the real space viewed from 
said player's viewpoint position; 

25 the first video sensing means for sensing a video of 
the real space viewed from a first viewpoint position 
which differ from said player's viewpoint position; 
the first video generation means for generating a 
video of the virtual object viewed from said first 

30 viewpoint position; 
and 

the first video composition means for compositing 
an augmented reality video viewed from said first 
viewpoint position on the basis of said videos of the 
35 real space and the virtual object viewed from said 
first viewpoint position. 

[0012] As a result, a video that reviews an operating 
state of a virtual object by the player in the augmented 
40 reality space from the objective viewpoint position can 
be generated and presented to a third party other than 
the player. 

[001 3] The augmented reality presentation apparatus 
as a preferred embodiment of the present invention fur- 
45 ther comprises the following feature described in claim 

2. 

[0014] That is, said augmented reality presentation 
means further comprises: 

so the second video sensing means for sensing a vid- 
eo of the real space viewed from said player's view- 
point position; 

the second video generation means for generating 
a video of the virtual object viewed from said play- 
55 e^s viewpoint position; 

the second video composition means for composit- 
ing an augmented reality video viewed from said 
player's viewpoint position on the basis of said vid- 
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eos of the real space and the virtual object viewed 

from said player's viewpoint position; 

and 

the display means for displaying to the player the 
augmented reality video viewed from said player's 
viewpoint position. 

[0015] The augmented reality presentation apparatus 
as a preferred embodiment of the present invention fur- 
ther comprises the following feature described in claim 
3. 

[0016] That is, said augmented reality presentation 
means further comprises: 

the second video generation means for generating 
a video of the virtual object viewed from said play- 
er's viewpoint position; 
and 

the display means for displaying to the player the 
video of the virtual object viewed from said player's 
viewpoint position on a display surface through 
which the player can visually see the real space. 

[001 7] The augmented reality presentation apparatus 
as a preferred embodiment of the present invention fur- 
ther comprises the following feature described in claim 
4. 

[0018] That is, information generation means for gen- 
erating information that pertains to rendering of the vir- 
tual object, and 

in that said first video generation means and said 
second video generation means generate videos of the 
virtual object using the information that pertains to ren- 
dering of the virtual object. 

[001 9] The augmented reality presentation apparatus 
as a preferred embodiment of the present invention fur- 
ther comprises the following feature described in claim 
5. 

[0020] That is, said information generation means 
generates information of an outer appearance of the vir- 
tual object and information of a position/posture of the 
virtual object as the information that pertains to render- 
ing of the virtual object. 

[0021 ] The augmented reality presentation apparatus 
as a preferred embodiment of the present invention fur- 
ther comprises the following feature described in claim 
6. 

[0022] That is, parameters of said first video sensing 
means are known, and 

said first video generation means generates the 
video of the virtual object viewed from said first view- 
point position in accordance with the known parameters. 
[0023] The augmented reality presentation apparatus 
as a preferred embodiment of the present invention fur- 
ther comprises the following feature described in-claim 
7. 

[0024] That is, some of parameters of said first video 
sensing means are variable, 



said apparatus further comprises measurement 
means for measuring changes of the parameters, 
and 

said first video generation means generates the vid- 
5 eo of the virtual object viewed from said first view- 
point position in accordance with the parameters 
measured by said measurement means. 

[0025] When the parameters of the objective view- 
10 point video sensing means, the objective viewpoint vid- 
eo generation means receives parameters from the 
measurement means, and generates an objective view- 
point video according to the received parameters. 
[0026] The augmented reality presentation apparatus 
15 as a preferred embodiment of the present invention fur- 
ther comprises the following feature described in claim 
8. 

[0027] That is, the parameters of said first video sens- 
ing means measured by said measurement means in- 
20 elude at least one of a viewpoint position/posture, and 
zoom ratio. 

[0028] The objective viewpoint video of a virtual object 
is generated in accordance with camera parameters 
(external parameters (viewpoint position/posture) and 

25 internal parameters (zoom ratio, aspect ratio, optical ax- 
. is central position , distortion ratio)) of the objective view- 
point video sensing means. The camera parameters 
measured by the measurement means preferably in- 
clude all parameters to be changed dynamically of those 

30 parameters. 

[0029] Other features and advantages of the present 
invention will be apparent from the following description 
taken in conjunction with the accompanying drawings, 
in which like reference characters designate the same 

35 or similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0030] The accompanying drawings, which are incor- 
40 porated in and constitute a part of the specification, il- 
lustrate embodiments of the invention and, together with 
the description, serve to explain the principles of the in- 
vention. 

45 f \g, 1 is a view showing the first embodiment; 

Fig. 2 is a block diagram showing the flow of proc- 
esses of the first embodiment; 
Fig. 3 is a diagram showing the generation process 
of a coordinate conversion matrix used to convert 

50 the object coordinate position of a virtual object into 
an image coordinate position viewed from a given 
viewpoint; 

Fig. 4 is a block diagram showing the flow of proc- 
esses of the third embodiment; 
55 Fig. 5 is a view showing a video presented to a play- 
er in the first to fourth embodiments; 
Fig. 6 is a view showing a video presented to a play- 
er in the fifth embodiment; 
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Fig. 7 is a block diagram showing the internal ar- 
rangement of an AR game apparatus; , 
Fig. 8 is a flow chart for generating a subjective 
viewpoint AR video and an objective viewpoint AR 
video; 

Fig. 9 is a diagram showing a camera system used 
in the fourth embodiment; 
Fig. 10 is a flow chart in the fourth embodiment; 
Fig. 11 is a flow chart in the fifth embodiment; 
Fig. 12 is a table showing camera parameters of a 
plurality of cameras in the fourth embodiment; 
Fig. 13 is a view for explaining an optical see- 
through HMD; 

Fig. 1 4 is a block diagram showing the flow of proc- 
esses of the second embodiment; 
Fig. 1 5 is a block diagram showing the sixth embod- 
iment; and 

Fig. 1 6 is a block diagram showing the seventh em- 
bodiment. 

Fig. 1 7 is a block diagram showing the flow of proc- 
esses of the tenth embodiment. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0031] Preferred embodiments of the present inven- 
tion will now be described in detail in accordance with 
the accompanying drawings. 

[First Embodiment] 

[0032] As one preferred embodiment of the present 
invention, this embodiment will explain an augmented 
reality presentation apparatus and method, which 
present a state in which a player is playing an AR (aug- 
mented reality) game in an AR space to a third party (to 
be referred to as a watcher hereinafter) other than the 
player. 

[0033] Fig. 1 shows this embodiment. 
[0034] Reference numeral 1 01 denotes a player who 
is playing an AR (augmented reality) game, and wears 
a head-mounted display (to be abbreviated as an HMD 
hereinafter) 1 07 on his or her head. Note that the HMD 
107 in this embodiment is of video see-through type 
which possesses video camera(s) inside or on it to cap- 
ture the video(s) from the same viewpoint of the player's 
eye(s). Since the video see-through HMD is known to 
those who are skilled in the art, a detailed description 
thereof will be omitted. 

[0035] Reference numeral 1 02 denotes virtual objects 
as characters which appear in the AR game. Each vir- 
tual object 1 02 is a three-dimensional virtual object ren- 
dered by computer graphics. The virtual objects 1 02 are 
generated by a method to be described later by an AR 
game apparatus 1 04. The player 1 01 can review a video 
obtained by superimposing the virtual objects 102 on a 
real space (to be referred to as an A R video hereinafter) 
from his or her subjective viewpoint by wearing the HMD 



1 07 on his or her head, and can consequently play the 
AR game. 

[0036] Reference numeral 103 denotes an objective 
viewpoint video sensing camera (to be simply referred 
to as a camera hereinafter) for sensing a state in which 
the player 101 is playing the AR game. In this embodi- 
ment, this camera 1 03 is fixed at a predetermined posi- 
tion and posture. A video sensed by the camera 103 (to 
be referred to as an objective viewpoint actually sensed 
video) is sent to the AR game apparatus 1 04 via a cable 
shown in Fig. 1 . Note that the sensed video is an actually 
sensed video, which does not include any images of the 
virtual objects 1 02. 

[0037] Reference numeral 104 denotes an AR game 
apparatus which controls the AR game, generates the 
videos of the virtual objects 102, and generates an AR 
video to be output to the HMD 107 and a display 106. 
Note that a video which the AR game apparatus 104 
outputs to the HMD 107 is an AR video (to be referred 
to as a subjective viewpoint AR video hereinafter) ob- 
tained by compositing an actually sensed video of the 
real space (to be referred to as a subjective viewpoint 
actually sensed video hereinafter) input from the HMD 
107, and videos of the virtual objects 102 viewed from 
the subjective viewpoint (to be referred to as subjective 
viewpoint virtual videos hereinafter). On the other hand, 
a video that the AR game apparatus 104 outputs to the 
display 106 is an AR video (to be referred to as an ob- 
jective viewpoint AR video hereinafter) obtained by 
compositing videos of the virtual objects 102 viewed 
from the objective viewpoint (to be referred to as objec- 
tive viewpoint virtual videos hereinafter), and the objec- 
tive viewpoint actually sensed video sensed by the cam- 
era 103. 

[0038] Reference numeral 105 denotes a table as a 
real object used as a stage in the AR game. 
[0039] Reference numeral 106 denotes a display 
which displays the objective viewpoint AR video gener- 
ated by the AR game apparatus 1 04 to present the play- 
ing state of the AR game by the player 101 to a third 
party other than the player 1 01 , as described above. 
[0040] Reference numeral 107 denotes the afore- 
mentioned HMD which displays the subjective viewpoint 
AR video generated by the AR game apparatus 104 on 
its display screen 501 shown in Fig. 5. Also, the HMD 
107 senses the subjective viewpoint actually sensed 
video viewed from the subjective viewpoint of the player 
101. The sensed video is sent to the AR game apparatus 
1 04 via a cable shown in Fig. 1 . 
[0041] The flow of processes until the AR game ap- 
paratus 104 displays the subjective and objective AR 
videos respectively on the HMD 107 and display 106 
based on the aforementioned arrangement will be ex- 
plained below using Fig. 2 that shows the flow of such 
processes. 

[0042] Reference numeral 201 denotes a game state 
manager incorporated in the AR game apparatus 104. 
The game state manager 201 manages the state of the 
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AR game (information that pertains to rendering of the 
virtual objects 1 02, the score of the player 1 01 , AR game 
round count, and the like) . Note that the information that 
pertains to rendering of the virtual objects 102 includes 
information of the outer appearance of each virtual ob- 
ject 1 02, and information of the position/posture of each 
virtual object 102 in the world coordinate system. 
[0043] The information of the outer appearance of 
each virtual object 1 02 is information that pertains to pol- 
ygons which build the virtual object 1 02, i.e., the number 
of polygons, the coordinate values of polygons, the 
colors of polygons, and the like. When each virtual ob- 
ject 102 has undergone texture mapping, the informa- 
tion of the outer appearance of each virtual object 1 02 
includes a texture size, texture file name, and the like. 
[0044] Reference numeral 204 denotes a camera pa- 
rameter measurement/management unit which meas- 
ures/manages camera parameters as parameters of the 
HMD 107 and camera 103. The camera parameters to 
be managed by the camera parameter measurement/ 
management unit 204 includes viewpoint position/pos- 
ture information as external parameters, and informa- 
tion of the field angle, focal length, distortion, and the 
like as internal parameters. The camera parameter 
measurement/management unit 204 stores the internal 
parameters of the HMD 107 as known information. At 
the same time, the unit 204 measures the external pa- 
rameters (information of the viewpoint position and pos- 
ture) of the HMD 1 07, and manages the camera param- 
eters of the HMD 1 07. Furthermore, the.camera param- 
eter measurement/management unit 204 manages 
camera parameters of the camera 103 as known infor- 
mation. 

[0045] Reference numeral 202 denotes an objective 
viewpoint video generator built in the AR game appara- 
tus 1 04. The objective viewpoint video generator 202 
generates objective viewpoint virtual videos on the basis 
of the information that pertains to rendering of the virtual 
objects 1 02, which is input from the game state manager 
201. 

[0046] Reference numeral 203 denotes an objective 
viewpoint video composition unit built in the AR game 
apparatus 1 04, which generates an objective viewpoint 
AR video by compositing objective viewpoint virtual vid- 
eos generated by the objective viewpoint video gener- 
ator 202, and an objective viewpoi nt actually sensed vid- 
eo input from the camera 1 03. 
[0047] Generation of the subjective viewpoint video 
will be explained below. 

[0048] The game state manager 201 updates the in- 
formation that pertains to rendering of the virtual objects 
102 as needed, and outputs the updated information to 
a subjective viewpoint video generator 21 2. The subjec- 
tive viewpoint video generator 212 generates videos of 
the virtual objects 102 (subjective viewpoint virtual vid- 
eo) viewed from the viewpoint position/posture of the 
HMD107, on the basis of the information that pertains 
to rendering of the virtual objects 102, which is input 



from the game state manager 201 , and the camera pa- 
rameters of the HMD 1 07, which are input from the cam- 
era parameter measurement/management unit 204. 
[0049] The generated subjective viewpoint virtual vid- 
s eo is output to a subjective viewpoint video composition 
unit 213. 

[0050] The subjective viewpoint video composition 
unit 213 receives from the HMD 107 the subjective view- 
point actually sensed video that the player 1 01 watches 
10 via the HMD 107. The subjective viewpoint video com- 
position unit 213 generates a subjective viewpoint AR 
video as a composite video of this subjective viewpoint 
actually sensed video, and the subjective viewpoint vir- 
tual videos input from the subjective viewpoint video 
15 generator 21 2, and outputs that subjective viewpoint AR 
video to the HMD 1 07. The subjective viewpoint AR vid- 
eo is displayed on the display screen 501 of the HMD 
107, and the player 101 plays the AR game while re- 
viewing this subjective viewpoint AR video. 
[0051 ] Generation of an objective viewpoint video will 
be explained below. 

[0052] The game state manager 201 updates the in- 
formation that pertains to rendering of the virtual objects 
102 as needed, and outputs the updated information to 
the objective viewpoint video generator 202. 
[0053] Since generation of the videos of the virtual ob- 
jects 102 is implemented by the same processes as 
those in generation of the subjective viewpoint video, a 
detailed description thereof will be omitted. 
[0054] The generated objective viewpoint virtual vid- 
eo is output to the objective viewpoint video composition 
unit 203. 

[0055] The objective viewpoint video composition unit 
203 receives an objective viewpoint actually sensed vid- 
eo from the camera 103. The objective viewpoint video 
composition unit 203 generates an objective viewpoint 
AR video as a composite video of this objective view- 
point actually sensed video, and the objective viewpoint 
virtual videos input from the objective viewpoint video 
generator 202, and outputs the objective viewpoint AR 
video to the display 106. This objective viewpoint AR 
video is displayed on the display 106, and the watcher 
can see the overall view of the AR game by reviewing 
this objective viewpoint AR video and can recognize the 
current situation of the AR game. 
[0056] The image coordinate values of the vertices of 
the polygons that build each virtual object 102 viewed 
from each viewpoint (HMD 107 or camera 103) can be 
computed using the coordinate conversion matrix Ml or 
M2 from the object coordinate system into the image co- 
ordinate system. The generation process of such coor- 
dinate conversion matrix will be explained below using 
the block diagram shown in Fig. 3. 
[0057] A coordinate conversion matrix Mm from the 
55 object coordinate system into a world coordinate system 
is computed on the basis of the information of the posi- 
tion/posture of each virtual object 102. Also, coordinate 
conversion matrices Mcl and Mc2 from the world coor- 
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dinate system into the corresponding camera coordi- 
nate systems are respectively computed on the basis of 
information of the positions/postures of respective view- 
points included in the camera parameters of the HMD 
107 and camera 103. Furthermore, coordinate conver- 
sion matrices Mdl and Md2 for implementing perspec- 
tive conversions from the corresponding camera coor- 
dinate systems into the image coordinate system are re- 
spectively computed on the basis of information such as 
the field angles, focal lengths, distortions, and the like 
included in the camera parameters of the HMD 1 07 and 
camera 103. The aforementioned coordinate conver- 
sion matrices are computed by the subjective viewpoint 
video generator 21 2 or objective viewpoint video gener- 
ator 202. 

[0058] The conversion matrix Ml from the object co- 
ordinate system into the image coordinate system of the 
HMD 107 is given by: 

M1 = MdlMdMm 

On the other hand, the conversion matrix M2 from the 
object coordinate system into the image coordinate sys- 
tem of the camera 1 03 is given by: 

M2 = Md2Mc2Mm 

These computations are made by the subjective view- 
point video generator 212 or objective viewpoint video 
generator 202. 

[0059] The present invention is not limit to these com- 
putations to make images. These computations are 
used, as factors that characterize subjective viewpoint 
and objective viewpoint. 

[0060] The internal arrangement and operation of the 
AR game apparatus 104 will be explained below using 
Fig. 7 that shows the internal block diagram of the AR 
game apparatus 1 04. 

[0061] Reference numeral 701 denotes a CPU, which 
executes a program code loaded onto a RAM 703. The 
CPU 701 also has an area for temporarily saving data 
during execution of a program. 
[0062] Reference numeral 702 denotes a ROM which 
stores setups upon and after starting up the AR game 
apparatus 104, and a startup program code. Also, the 
ROM 702 stores character codes and the like used to 
display on the display screen 501 a score or the like out- 
put to the HMD 107 during the AR game. 
[0063] Reference numeral 703 denotes a RAM which 
stores a program code of the AR game loaded from a- 
floppy disk, CD-ROM, or the like as an external storage 
medium, polygon and texture data that build each virtual 
object 102, and the like. Note that the game state man- 
ager 201 refers to and manages polygon (texture) data 
that build each virtual object 1 02, which are stored in the 
RAM 703. 



[0064] Reference numeral 704 denotes an interface 
(to be abbreviated as an l/F hereinafter), which is used 
to connect the AR game apparatus 104 to an external 
apparatus. The HMD 1 07, camera 1 03, and display 1 06 
5 are all connected to the AR game apparatus 104 via this 
l/F 704. 

[0065] Reference numeral 705 denotes a console 
which comprises a keyboard and a pointing device such 
as a mouse or the like. This console 705 allows the user 
10 to input setup commands of the AR game apparatus 
104, and those for peripheral devices connected to the 
AR game apparatus 104. 

[0066] Reference numeral 706 denotes a bus that 
connects the aforementioned units. Note that the game 
15 state manager 201 , objective viewpoint video generator 
202, objective viewpoint video composition unit 203, 
subjective viewpoint video generator 212, subjective 
viewpoint video composition unit 213, and camera pa- 
rameter measurement/management unit 204 are con- 
20 nected to this bus 706, and are controlled via the bus 
706 on the basis of the program code loaded onto the 
RAM 703, as described above. 
[0067] When the CPU 701 executes the program 
code loaded onto the RAM 703, the game state manag- 
25 er 201 , objective viewpoint video generator 202, objec- 
tive viewpoint video composition unit 203, subjective 
viewpoint video generator212, subjective viewpoint vid- 
eo composition unit 213, and camera parameter meas- 
urement/management unit 204 execute the aforemen- 
30 tioned processes, and output the subjective and objec- 
tive viewpoint AR videos to the HMD 107 and display 
106, respectively. The flow chart of this program code 
is shown in Fig. 8, and will be described below. 
[0068] In step S801 , the camera parameter measure- 
as ment/management unit 204 updates the information of 
the viewpoint position/posture of the HMD 107. 
[0069] In step S802, the game state manager201 up- 
dates the game state (information that pertains to ren- 
dering of the virtual objects 102). 
40 [0070] In step S803, the objective and subjective 
viewpoint video generators 202 and 212 respectively 
generate objective and subjective viewpoint videos. 
[0071] In step S804, the objective and subjective 
viewpoint video composition units 203 and 213 respec- 
ts tively receive actually sensed videos sensed by the 
HMD 107 and 103. 

[0072] In step S805, the objective and subjective 
viewpoint video composition units 203 and 213 respec- 
tively generate objective and subjective viewpoint AR 
50 videos. 

[0073] In step S806, the objective and subjective 
viewpoint video composition units 203 and 213 respec- 
tively output the objective and subjective viewpoint AR 
videos to the display 106 and HMD 107. 
55 [0074] The aforementioned processes are repeated 
until the AR game ends. 

[0075] With the aforementioned arrangement of the 
apparatus, augmented reality presentation method, and 
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program code, an objective viewpoint AR video can be 
presented to a third party other than the player 101 in 
the AR game using the video see-through HMD 107. 

[Second Embodiment] 

[0076) In the first embodiment, the HMD 107isofvid- 
eo see-through type. However, if the HMD 107 is of op- 
tical see-through type, the player 101 can still play the 
AR game. 

[0077] Fig. 13 shows an optical see-through HMD 
1301. Note that Fig. 13 schematically illustrates the 
HMD 1301, and the present invention is not limited to 
the size and shape shown in Fig. 13. 
[0078] Reference numeral 1301 denotes an optical 
see-through HMD; and 1 302, the eye of the player 1 01 . 
[0079] On the display screen 501 , only videos of the 
virtual objects 102 (subjective viewpoint virtual videos) 
are displayed. On the other hand, a video of a real space 
is seen behind the display screen 501 when viewed from 
the position of the eye 1302. Hence, the player can re- 
view videos of the virtual objects 1 02 and the real space 
seen behind the display screen 501 when viewed from 
the position of the eye 1302 to overlap each other by 
observing the display screen 501 . 
[0080] Fig. 14 is a block diagram showing the flow of 
processes of this embodiment. 
[0081] A video output from the AR game apparatus 
104 to the HMD 1301 is a subjective viewpoint virtual 
video alone, as described above. Hence, the HMD 1 301 
and subjective viewpoint video generator 212 are elec- 
trically connected via a cable, and a subjective viewpoint 
virtual video which is generated by the subjective view- 
point video generator 212 on the basis of the position 
and posture of the HMD 1 301 is sent from the subjective 
viewpoint video generator 212 to the HMD 1301 . Note 
that this embodiment has no subjective viewpoint video 
composition unit 213 in the first embodiment. 
[0082] Also, the method of generating an objective 
viewpoint AR video is the same as that which has been 
explained in the first embodiment. 
[0083] The internal block diagram of the AR game ap- 
paratus 104 in this embodiment is substantially the 
same as that shown in Fig. 7, except that the objective 
viewpoint video composition unit 213 is removed from 
the arrangement shown in Fig. 7. 
[0084] In the flow chart of this embodiment, steps 
S804, S805, and S806 in Fig. 8 are rewritten as follows. 
More specifically, upon generating a subjective view- 
point AR video (only videos of the virtual objects 102 
based on the position and posture of the HMD 1301 in 
this embodiment) to be displayed on the HMD 1301, 
steps S804 and S805 are skipped. That is, in step S804 
the objective viewpoint video composition unit 203 re- 
ceives an actually sensed video sensed by the camera 
103. In step SB05, the objective viewpoint video com- 
position unit 203 generates an objective viewpoint AR 
video. In step S806, the objective viewpoint video com- 



position unit 203 outputs the objective viewpoint AR vid- 
eo to the display 1 06, and the subjective viewpoint video 
generator 21 2 outputs a subjective viewpoint virtual vid- 
eo to the HMD 1301. 

5 [0085] The flow chart obtained by modifying the con- 
tents of Fig. 8 as described above is that in this embod- 
iment, and this embodiment is controlled by a program 
code according to this modified flow chart. 
[0086] With the aforementioned arrangement of the 

10 apparatus, augmented reality presentation method, and 
program code, an objective viewpoint AR video can be 
presented to a third party other than the player 101 in 
the AR game using the optical see-through HMD 1301 . 

18 [Third Embodiment] 

[0087] In the first and second embodiments, the cam- 
era parameters of the camera 1 03 are fixed. That is, an 
objective viewpoint AR video is generated based on the 
20 camera parameters of the camera 1 03 which is fixed in 
position. The camera parameter data of the camera 1 03 
are stored as permanent values in the RAM 703 of the 
AR game apparatus 1 04. 

[0088] A case will be examined below wherein the 
25 viewpoint position, posture, and zooming ratio of the 
camera 103 are changed in real time to those that the 
player or a third party other than the player 1 01 (watcher 
or operator) wants. That is, a case will be examined be- 
low wherein the camera parameters of the camera 1 03 
30 are changed in real time. Note that this embodiment us- 
es a video see-through HMD as in the first embodiment. 
However, the HMD that can be used in this embodiment 
is not limited to the video see-through type, but an opti- 
cal see-through HMD may be used, as can be seen from 
35 the description of the second embodiment and this em- 
bodiment. 

[0089] When the position, posture, and zooming ratio 
of the camera 103 are to be changed in real time, a 
measurement means as a means for measuring the po- 

40 sition, posture, and zooming ratio of the camera 103 
must be added to the first embodiment. 
[0090] Fig. 4 is a block diagram showing the flow of 
processes in this embodiment. The flow of processes in 
this embodiment will be described below using Fig. 4. 

45 [0091] Like in the first embodiment, in this embodi- 
ment the camera parameter measurement/manage- 
ment unit 204 measures and manages the camera pa- 
rameters of the HMD 107, and holds some internal pa- 
rameters of the camera 103 as known information. Un- 

so like in the first embodiment, in this embodiment the cam- 
era parameter measurement/management unit 204 
controls a sensor (not shown) attached to the camera 
1 03 to measure the position, posture, and zooming ratio 
of the camera 103. 

55 [0092] When the player 101 has changed the position, 
posture, and zooming ratio of the camera 1 03 via an in- 
terface (not shown), the camera parameter measure- 
ment/management unit 204 measures the position, pos- 
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ture, and zooming ratio of the camera 103. The meas- 
urement result data (the position, posture, and zooming 
ratio data of the camera 1 03) by the camera parameter 
measurement/management unit 204, and the known in- 
ternal parameters are output to the objective viewpoint 
video generator 202. The objective viewpoint video gen- 
erator 202 generates an objective viewpoint virtual vid- 
eo on the basis of the input camera parameters of the 
camera 1 03 in the same manner as in the process in the 
first embodiment. Other processes are the same as 
those in the first embodiment. 

[0093] A control program code of the interface that 
changes the position, posture, and zooming ratio of the 
camera 1 03, and a control program code of the camera 
parameter measurement/managementunit 204 are 
stored in the RAM 703 of the AR game apparatus 104. 
[0094] The flow chart of this embodiment is the same 
as that shown in Fig. 8. However, in this embodiment 
the camera parameter measurement/management unit 
204 updates the information of the viewpoint position 
and posture of the HMD 1 07 and the information of the 
viewpoint position, posture, and the zooming ratio of the 
camera 103 in step S801 . 

[0095] Internal parameter (zooming ratio) is fixed, po- 
sition and posture may be variable. With the aforemen- 
tioned arrangement of the apparatus, augmented reality 
presentation method, and program code, an objective 
viewpoint AR video can be generated based on data 
from the camera whose position and posture change. 

[Fourth Embodiment] 

[0096] In the first to third embodiments, only one cam- 
era is set. Alternatively, when a plurality of cameras are 
set and the camera to be used is switched, objective 
viewpoint AR videos from a plurality of positions and 
postures can be displayed on the display 106. 
[0097] Fig. 9 shows a camera system in this embod- 
iment. The camera system will be explained below using 
Fig. 9. Note that this embodiment sets three cameras. 
[0098] Reference numerals 901 a, 901 b, and 901 c de- 
note cameras (cameras 901a, 901b, and 901c), the 
camera parameters (positions, postures, field angles, 
and the like) of which are fixed. The functions of these 
cameras are the same as those in the first and second 
embodiments. The camera parameters of these three 
cameras are set, as shown in Fig. 12, and these three 
sets of data are stored in the RAM 703 of the AR game 
apparatus 1 04 and are managed by the camera param- 
eter measurement/management unit 204. 
[0099] Note that reference numerals 1201, 1202, and 
1 203 in Fig. 1 2 denote (selection information) tags cor- 
responding to the cameras 901 a, 901 b, and 901 c, which 
are used to select the camera to be used. 
[0100] Reference numeral 902 denotes a camera 
switching device, which has a function of a switcher that 
sends only a signal (actually sensed video of a real 
space sensed by the selected camera) from the camera 



selected by a selector 903 to the video composition unit 
203 via the switching device 902. 
[0101] Reference numeral 903 denotes a selector 
which has the aforementioned function, and has three 

5 buttons (buttons A, B, and C) corresponding to the three 
cameras. When the user selectively presses one of 
these buttons, he or she can select the camera to be 
used. The selector 903 outputs selection information in- 
dicating the camera selected to the switching device 902 

10 and camera parameter measurement/management unit 
204. Note that the cameras 901a, 901b, and 901c are 
respectively selected by pressing buttons A, B, and C 
in Fig. 9 as the camera to be used. 
[0102] The camera parameter measurement/man - 

15 agement unit 204 selects the tag (one of the tags 1201 , 
1202, and 1203) of the camera mentioned above in ac- 
cordance with the camera selection information input 
from the selector 903, and outputs the camera param- 
eters of the selected camera to the objective viewpoint 

20 video generator 202. For example, when the camera 
901b is used, the camera parameters of the camera 
901b held by the tag 1202 are output to the objective 
viewpoint video generator 202 upon depression of but- 
. ton B. 

25 [0103] Fig. 1 0 is a flow chart showing the process of 
this embodiment. 

[0104] It is checked in step S1 001 if one of the buttons 
of the selector 903 has been pressed. Note that this 
process is repeated until one of the buttons is pressed. 
30 [0105] In steps S1002, S1003, and S1 004, the button 
pressed is determined. Assuming that button B has 
been pressed, the flow advances from step S1 003 to 
step S1006. 

[0106] In steps S1 005, S1 006, and S1 007, an actually 
35 sensed video sensed by the camera corresponding to 
the pressed button is sent to the objective viewpoint vid- 
eo composition unit 203 via the switching device 902. 
Also, the selection information of the camera corre- 
sponding to the pressed button is sent to the camera 
40 parameter measurement/management unit 204. 

[0107] A program code according to the aforemen- 
tioned flow chart is stored in the internal memory (not 
shown) of the switching device 902. 
[0108] Camera switching may be automatically per- 
45 formed in accordance with the progress of a game. In 
this case, the selector 903 is omitted, and the game 
state manager 201 outputs camera selection informa- 
tion to the camera switching device 902 and the camera 
parameter measurement/management unit 204. Cam- 
50 era switching can employ a method of switching camer- 
as every predetermined time interval or a method of 
switching cameras in accordance with the progress of 
a scenario by presetting a camera having an appropriate 
camera angle every time the scenario progresses. 
55 [0109] In this embodiment, the camera parameters of 
the three cameras are fixed. Alternatively, when the 
camera parameters of these cameras change in real 
time, the aforementioned arrangement of the apparatus, 
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augmented reality presentation method, and program 
code can be applied. In this case, such application can 
be implemented by connecting the camera system of 
this embodiment to the AR game apparatus 1 04 in place 
of the camera 103 in the third embodiment. 
[0110] With the aforementioned arrangement of the 
apparatus, augmented reality presentation method, and 
program code, a plurality of objective viewpoint AR vid- 
eos sensed by a plurality of cameras can be displayed 
on the display 106. 

[Fifth Embodiment] 

[01 1 1 J In the first to fourth embodiments, an objective 
viewpoint AR video is presented to a third party other 
than the player via the display 1 06 by outputting it to the 
display 106. Alternatively, the objective viewpoint AR 
video may be presented to the player 101. That is, a 
display area 601 shown in Fig. 6 is assured on the dis- 
play screen 501 of the HMD (which can be of either vid- 
eo or optical see-through type) that the player 101 
wears, and the objective viewpoint AR video is dis- 
played there. In Fig. 6, a video displayed on this display 
screen 501 will be referred to as an augmented video 
hereinafter. 

[0112] In order to generate this augmented video, a 
program code for setting the display area 601 on the 
display screen 501 , and writing an objective viewpoint 
AR video on that display area 601 is stored in the RAM 
703 of the AR game apparatus 104 in addition to the 
program code according to the flow chart shown in Fig. 
8. As a result, by executing this program code, the aug- 
mented video can be displayed on the display screen 
501. 

[01 1 3] Fig. 1 1 is a flow chart of the program code for 
writing the objective viewpoint AR video on the display 
area 601 mentioned above. 

[0114] It is checked in step S1101 if the display area 
601 is assured. Selection as to whether or not the dis- 
play area 601 is assured can be implemented by pro- 
viding this selection switch to an operation device (not 
shown) used when the player 101 plays the AR game. 
Or such selection can be implemented by inputting a 
command indicating whether or not the display area 601 
is assured from the console 705. 
[0115] In step S1102, the display position of the dis- 
play area 601 is input. The display position is input from 
the console 705. Alternatively, the player 101 may input 
the display position using the aforementioned operation 
device. 

[01 16] In step S11 03, the size of the display area 601 
is input. The size is input from the console 705. Alterna- 
tively, the player 1 01 may input the size using the afore- 
mentioned operation device. 

[01 17] In step S1 1 04, the display area 601 , the setups 
of which have been determined in steps S1102 and 
S1103, is assured on the display screen 501 . 
[0118] In step S1105, the objective viewpoint AR vid- 
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eo generated by the objective viewpoint video compo- 
sition unit 203 is rendered on the display area 601 . As 
a consequence, the augmented video can be generat- 
ed. 

5 [01 1 9] Note that this augmented video may be output 
to the display 106. 

[0120] Selection of whetherthe display area 601 isset 
and the display position and size of the display area 601 
may be automatically set/changed not by the player but 
10 in accordance with the progress of the game. In this 
case, the game state manager 201 determines these 
parameters. 

[0121] The display area 601 may be the entire area 
of the display screen 501 . 
15 [0122] With the aforementioned arrangement of the 
apparatus, augmented reality presentation method, and 
program code, both the subjective and objective view- 
point AR videos can be presented to the player. 

20 [Sixth Embodiment] 

[0123] A plurality of players 101 may join the AR 
game. In this case, subjective viewpoint AR videos from 
individual subjective viewpoints must be provided to the 

25 individual players. Fig. 15 shows the internal arrange- 
ment of the AR game apparatus for providing subjective 
viewpoint AR videos from the players' subjective view- 
points to the individual players. Note that the camera 
103 is fixed in position, and the processes that pertain 

30 to the camera 1 03 and display are the same as those in 
the first embodiment. 

[0124] Fig. 15 shows the internal arrangement for 
three players. Hence, the AR game apparatus 1 04 com- 
prises HMDs 107A, 107B, and 107C, subjective view- 

35 point video composition units 213A, 213B, and 213C, 
and subjective viewpoint video generators 21 2A, 21 2B, 
and 21 2C in correspondence with three players a, b, and 
c. Three players a, b, and c respectively wear the HMDs 
107A, 107B, and 107C. This embodiment uses a video 

40 see-through HMD, but an optical see-through HMD may 
be used. In this case, the subjective viewpoint video 
composition units for the three HMDs can be omitted. 
[0125] When the AR game starts, the HMDs, subjec- 
tive viewpoint video generators, and subjective view- 

45 point video composition units execute the same proc- 
esses as those described in the first embodiment, and 
subjective viewpoint AR videos generated for the indi- 
vidual players are output to the HMDs 107A, 107B, and 
1 07C that the players wear. 

so [0126] Note that the aforementioned arrangement of 
the apparatus and augmented reality presentation 
method are not limited to three players, as is obvious 
from the above description. 

[0127] With the aforementioned arrangement of the 
55 apparatus and augmented reality presentation method, 
subjective viewpoint AR videos can be provided to a plu- 
rality of players. 
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[Seventh Embodiment] 

[0128] In the fourth embodiment, a plurality of camer- 
as 103 are set, but only one display 106 for displaying 
an objective viewpoint AR video is used. 
[01 29] However, the number of displays 1 06 is not lim- 
ited to one, but a plurality of displays 1 06 may be used. 
That is, when objective viewpoint AR videos from the 
respective cameras 103 are separately displayed on a 
plurality of displays 1 06, the objective viewpoint AR vid- 
eos from all the cameras 1 03 can be presented to a third 
party other than the player 101 using a plurality of dis- 
plays 1 06. 

[0130] In this embodiment, a case will be described 
below wherein a plurality of cameras 1 03 and a plurality 
of displays 106 are set. Fig. 16 shows the internal ar- 
rangement of the AR game apparatus 1 04 for displaying 
objective viewpoint AR videos from a plurality of camer- 
as 103 on a plurality of displays 106. 
[0131] In Fig. 16, three cameras 103A, 103B, and 
103C are set, and displays 106A, 106B, and 106C for 
respectively displaying objective viewpoint AR videos 
from the cameras 103A, 103B, and 103C are set. Also, 
the AR game apparatus 104 comprises objective view- 
point video generators 202A, 202B, and 202C, and ob- 
jective viewpoint video composition units 203A, 203B, 
and 203C corresponding to the displays 106A, 106B, 
and 1 06C so as to generate objective viewpoint AR vid- 
eos corresponding to the displays 106A, 106B, and 
106C. 

[0132] When the AR game starts, the cameras, ob- 
jective viewpoint video generators, and objective view- 
point video composition units execute the same proc- 
esses as those in the first embodiment, and objective 
viewpoint AR videos from the cameras 1 03A to 1 03C 
are displayed on the displays 1 06A to 1 06C correspond- 
ing to these cameras 103A to 103C. 
[0133] Note that the aforementioned arrangement of 
the apparatus and augmented reality presentation 
method are not limited to three displays 106, as is obvi- 
ous from the aforementioned description. 
[0134] With the aforementioned arrangement of the 
apparatus and augmented reality presentation method, 
all objective viewpoint AR videos from a plurality of cam- 
eras 1 03 can be presented to a third party other than 
the player 101 using a plurality of displays 106. 

[Eighth Embodiment] 

[0135] An apparatus that offers an AR video to the 
player 1 01 may be other than the game apparatus. That 
is, an AR apparatus which has substantially the same 
internal arrangement as that of the AR game apparatus 
1 04 and stores a program code for implementing shop- 
ping in an AR space in place of a game program code 
may be used in place of the AR game apparatus 104. 
As a result, the player 1 01 can browse among presented 
products in shopping by observing a subjective view- 



point AR video from the AR apparatus, and that state is 
displayed on the display 106 as an objective viewpoint 
AR video. 

[0136] Note that these subjective and objective view- 
5 point AR videos can be generated in the same manner 
as in the above embodiments. 

[Ninth Embodiment] 

10 [01 37] When the camera parameters such as the po- 
sition, posture, zoom ratio, and the like of the camera 
change, they may be estimated from an actually sensed 
video input from the camera 1 03 in place of the sensor. 
In this case, the camera parameter measurement/man - 

is agement unit 204 receives an actually sensed video, 
and estimates camera parameters by a known method. 
In consideration of estimation errors, the camera param- 
eters measured by the sensor and those estimated by 
the camera parameter measurement/management unit 

20 204 may be averaged to determine camera parameters 
to be used finally. 

(Tenth Embodiment] 

25 [0138] The generated objective viewpoint AR video 
can be not only displayed on the display 106 but also 
printed on a medium such as paper or afilm. In this case, 
a print controller 301 (print control apparatus 301) and 
a printer 302 are added to the arrangement shown in 

30 each of the first to seventh embodiments. The 1 0th em- 
bodiment will be described with reference to Fig. 1 7 by 
exemplifying the arrangement of the first embodiment. 
[0139] In the 10th embodiment, an objective view- 
point AR video output from an objective viewpoint video 

35 composition unit 203 is output to the print controller 301 
as well as the display 1 06. The print controller 301 stores 
the objective viewpoint AR video input at this time point 
and outputs the stored image to the printer 302 in ac- 
cordance with a command input from a console 705. 

40 The printer 302 prints the input image on paper. 

[0140] As in the seventh embodiment, when a plural- 
ity of objective viewpoint video composition units 203 
are used, the print controller 301 receives objective 
viewpoint AR videos respectively output from the objec- 
ts tive viewpoint video composition units 203. In this case, 
in accordance with a command input from the console 
705 the print controller 301 selects an objective view- 
point AR video having a specific viewpoint, stores the 
objective viewpoint AR video input at this time point, and 

so outputs the stored image to the printer 302. It is possible 
to print the subjective viewpoint AR video of each em- 
bodiment described above in accordance with the same 
processing as described above. In this case, a subjec- 
tive viewpoint video composition unit 213 outputs the 

55 subjective viewpoint AR video to an HMD 1 07 and at the 
same time to the print controller 301 . 
[0141] It is also possible to simultaneously print ob- 
jective viewpoint AR videos viewed from a plurality of 
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viewpoint positions, objective viewpoint AR videos 
viewed from the same viewpoint at different times, or 
subjective viewpoint A R videos of the respective players 
on one paper sheet. In this case, the print controller 301 
selects a viewpoint in accordance with a command input 
from the console 705 and stores the AR video viewed 
from the selected viewpoint. The print controller 301 de- 
termines a layout for printing a plurality of stored images 
on one paper sheet in accordance with a command input 
from the console 705 and outputs them to the printer 
302. 

[0142] The command may be automatically input to 
the print controller 301 in accordance with the progress 
of a game without using the console 705. In this case, 
a game state manager 201 transmits a command to the 
print controller 301 . The command may be transmitted 
at a fixed timing preset in accordance with the progress 
of the game or a dynamic timing corresponding to the 
progress of the game such that the distance between 
the player and a virtual object 1 02 becomes a predeter- 
mined interval or less. 

[01 43] The camera 1 03 (or camera 901 ) can be locat- 
ed at an arbitrary position. The third party can easily 
grasp the state of the game when the camera is located 
at a position where the entire AR space can be observed 
from the bird-view position or a position where the upper 
half image (so-called bust shot) of the player can be 
sensed from the front. 

[0144] As described above, according to the present 
invention, a video of the AR game viewed from the sub- 
jective viewpoint of the player can be generated, and, 
simultaneously, a video of the AR game viewed from an 
objective viewpoint to see the overall view of the game 
field or a video of the AR game viewed from a viewpoint 
the watcher wants can be generated. 
[0145] As many apparently widely different embodi- 
ments of the present invention can be made without de- 
parting from the spirit and scope thereof, it is to be un- 
derstood that the invention is not limited to the specific 
embodiments thereof except as defined in the append- 
ed claims. 

[0146] A game state manager (201) manages the 
state of an AR game (information that pertains to ren- 
dering of each virtual object (1 02), the score of a player 

(101) , the AR game round count, and the like). An ob- 
jective viewpoint video generator (202) generates a vid- 
eo of each virtual object (102) viewed from a camera 
(103). An objective viewpoint video composition unit 
(203) generates a composite video of the video of the 
virtual object (102) and an actually sensed video, and 
outputs itto a display (1 06). A subjective viewpoint video 
generator (212) generates a video of the virtual object 

(1 02) viewed from an HMD (1 07). A subjective viewpoint 
video composition unit (21 3) generates a composite vid- 
eo of the video of the virtual object (1 02) and an actually 
sensed video, and outputs it to the HMD (107). 



Claims 

1. An augmented reality presentation apparatus for 
superimposing a virtual object in a real space, char- 
5 acterized by comprising: 

augmented reality presentation means for su- 
perimposing the virtual object viewed from a 
player's viewpoint position in the real space 

10 viewed from said player's viewpoint position; 

the first video sensing means for sensing a vid- 
eo of the real space viewed from a first view- 
point position which differ from said player's 
viewpoint position; 

15 the first video generation means for generating 

a video of the virtual object viewed from said 
first viewpoint position; 
and 

the first video composition means for compos- 
20 iting an augmented reality video viewed from 

said first viewpoint position on the basis of said 
videos of the real space and the virtual object 
viewed from said first viewpoint position. 

25 2. The apparatus according to claim 1 , characterized 
in that said augmented reality presentation means 
further comprises: 

the second video sensing means for sensing a 
30 video of the real space viewed from said play- 

er's viewpoint position; 

the second video generation means for gener- 
ating a video of the virtual object viewed from 
said player's viewpoint position; 

35 the second video composition means for com- 

positing an augmented reality video viewed ■ 
from said player's viewpoint position on the ba- 
sis of said videos of the real space and the vir- 
tual object viewed from said player's viewpoint 

40 position; 
and 

the display means for displaying to the player 
the augmented reality video viewed from said 
player's viewpoint position. 

45 

3. The apparatus according to claim 1 , characterized 
in that said augmented reality presentation means 
further comprises: 

so the second video generation means for gener- 

ating a video of the virtual object viewed from 
said player's viewpoint position; 
and 

the display means for displaying to the player 
55 the video of the virtual object viewed from said 

player's viewpoint position on a display surface 
through which the player can visually see the 
real space. 
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4. The apparatus according to any one of claim 1 to 3, 
characterized by further comprising information 
generation means for generating information that 
pertains to rendering of the virtual object, and 

in that said first video generation means and 
said second video generation means generate vid- 
eos of the virtual object using the information that 
pertains to rendering of the virtual object. 

5. The apparatus according to claim 4, characterized 
in that said information generation means gener- 
ates information of an outer appearance of the vir- 
tual object and information of a position/posture of 
the virtual object as the information that pertains to 
rendering of the virtual object. 

6. The apparatus according to claim 1 or 4, character- 
ized in that parameters of said first video sensing 
means are known, and 

said first video generation means generates 
the video of the virtual object viewed from said first 
viewpoint position in accordance with the known pa- 
rameters. 

7. The apparatus according to claim 1 or 4, character- 
ized in that some of parameters of said first video 
sensing means are variable, 



first viewpoint position using parameters of the 
first video sensing means selected by said se- 
lection means. 

5 10. An augmented reality presentation method for su- 
perimposing a virtual object in a real space, char- 
acterized by comprising: 

augmented reality presentation step of super- 
imposing the virtual object viewed from a play- 
er's viewpoint position in the real space viewed 
from said player's viewpoint position; 
the first video sensing step of sensing a video 
of the real space viewed from a first viewpoint 
position which differ from said player's view- 
point position; 

the first video generation step of generating a 
video of the virtual object viewed from said first 
viewpoint position; 
and 

the first video composition step of compositing 
an augmented reality video viewed from said 
first viewpoint position on the basis of said vid- 
eos of the real space and the virtual object 
viewed from said first viewpoint position. 

1 1 . The method according to claim 1 0, characterized in 
that the augmented reality presentation step further 
comprises: 

the second video sensing step of sensing a vid- 
eo of the real space viewed from said player's 
viewpoint position; 

the second video generation step of generating 
a video of the virtual object viewed from said 
player's viewpoint position; 
the second video composition step of compos- 
iting an augmented reality video viewed from 
said player's viewpoint position on the basis of 
said videos of the real space and the virtual ob- 
ject viewed from said player's viewpoint posi- 
tion; 
and 

the display step of displaying to the player the 
augmented reality video viewed from said play- 
er's viewpoint position. 

12. The method according to claim 1 0, characterized in 
that the augmented reality presentation step further 
comprises: 

the second video generation step of generating 
a video of the virtual object viewed from said 
player's viewpoint position; 
and 

the display step of displaying to the player the 
video of the virtual object viewed from said play- 
er's viewpoint position on a display surface 



said apparatus further comprises measure- 
ment means for measuring changes of the pa- 30 
rameters, and 

said first video generation means generates the 
video of the virtual object viewed from said first 
viewpoint position in accordance with the pa- 
rameters measured by said measurement 35 
means. 

8. The apparatus according to claim 7, characterized 
in that the parameters of said first video sensing 
means measured by said measurement means in- *o 
elude at least one of a viewpoint position/posture, 
and zoom ratio. 

9. The apparatus according to claim 1 , characterized 
in that when a plurality of first video sensing means 45 
equivalent to said first video sensing means are 
present, 

said apparatus further comprises selection 
means for receiving a plurality of videos of the so 
real space from said first viewpoint position 
from the plurality of first video sensing means, 
and outputting a video of the real space viewed 
from said first viewpoint position input from one 
selected first video sensing means to said first ss 
video composition means, and 
said first video composition means generates 
a video of the virtual object viewed from said 
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through which the player can visually see the 
real space. 

13. The method according to any one of claim 10 to 12, 
characterized by further comprising the information 
generation step of generating information that per- 
tains to rendering of the virtual object, 

and 

in that in said first video generation step and 
said second video generation step, videos of the vir- 
tual object are generated using the information that 
pertains to rendering of the virtual object. 

1 4. The method according to claim 1 3, characterized in 
that said information generation step includes the 
step of generating information of an outer appear- 
ance of the virtual object and information of a posi- 
tion/posture of the virtual object as the information 
that pertains to rendering of the virtual object. 

15. The method according to claim 10 or 13, character- 
ized in that parameters of means for sensing said 
first viewpoint video are known, and 

said first video generation step includes the 
step of generating the video of the virtual object 
viewed from said first viewpoint position in accord- 
ance with the known parameters. 

16. The method according to claim 1 0 or 13, 

characterized 
in that some of parameters of means for sensing a 
video viewed from said first viewpoint position are 
variable, 

said method further comprises the measure- 
ment step of measuring changes of the param- 
eters, 
and 

said first video generation step includes the 
step of generating the video of the virtual object 
viewed from said first viewpoint position in ac- 
cordance with the parameters measured in the 
measurement step. 

17. The method according to claim 16, characterized in 
that the parameters of the means for sensing a vid- 
eo viewed from said first viewpoint position meas- 
ured in the measurement step include at least one 
of a viewpoint position/posture, and zoom ratio. 

18. The method according to claim 10, characterized in 
that when a plurality of means for sensing a video 
viewed from said first viewpoint position are 
present, 

said method further comprises the selection 
step of receiving a plurality of videos of the real 
space viewed from a first viewpoint position 



from the plurality of means for sensing a video 
viewed from said first viewpoint position, and 
outputting the video of the real space viewed 
from a first viewpoint position input from one se- 

5 lected means for sensing a video of said first 

viewpoint position to means for compositing a 
first viewpoint video, and 
said first video composition step includes the 
step of generating a video of the virtual object 

10 viewed from said first viewpoint position using 

parameters of the means for sensing a video 
viewed from a first viewpoint position selected 
in the selection step. 

15 19. A storage medium storing a program code for su- 
perimposing a virtual object in a real space when 
said program code is loaded by a computer, 
characterized by comprising: 

20 a program code of the augmented reality pres- 

entation step of superimposing the virtual ob- 
ject viewed from a player's viewpoint position 
in the real space viewed from said player 1 view- 
point position; 

25 a program code of the first video sensing step 

of sensing a video of the real space viewed from 
a first viewpoint position which differ from said 
player's viewpoint position; 
a program code of the first video generation 

30 step of generating a video of the virtual object 

viewed from said first viewpoint position; 
and 

a program code of the first video composition 
step of compositing an augmented reality video 
35 viewed from said first viewpoint position on the 

basis of said videos of-the real space and the 
virtual object viewed from said first viewpoint 
position. 

40 20. The medium according to claim 1 9, characterized 
in that the program code of the augmented reality 
presentation step further comprises: 

a program code of the second video sensing 
45 step of sensing a video of the real space viewed 

from said player's viewpoint position; 
a program code of the second video generation 
step of generating a video of the virtual object 
viewed from said player's viewpoint position; 
so a program code of the second video composi- 

tion step of compositing an augmented reality 
video viewed from said player's viewpoint posi- 
tion on the basis of said videos of the real space 
and the virtual object viewed from said player's 
55 viewpoint position; 

and 

a program code of the display step of displaying 
to the player the augmented reality video 



13 



7/16/2007, EAST Version: 2.1.0.14 



25 



EP1 117 074 A2 



26 



viewed from said player's viewpoint position. 

21. The medium according to claim 19, characterized 
in that the program code of the augmented reality 
presentation step further comprises: 

a program code of the second video generation 
step of generating a video of the virtual object 
viewed from said player's viewpoint position; 
and 

a program code of the display step of displaying 
to the player the video of the virtual object 
viewed from said player's viewpoint position on 
a display surface through which the player can 
visually see the real space. 

22. The medium according to any one of claim 19 to 21 , 
characterized by further comprisi ng a program code 
of the information generation step of generating in- 
formation that pertains to rendering of the virtual ob- 
ject, 

and 

in that in the program codes of said first video 
generation step and said second video generation 
step, videos of the virtual object are generated us- 
ing the information that pertains to rendering of the 
virtual object. 

23. The medium according to claim 22, characterized 
in that the program code of said information gener- 
ation step includes the step of generating informa- 
tion of an outer appearance of the virtual object and 
information of a position/posture of the virtual object 
as the information that pertains to rendering of the 
virtual object. 

24. The medium according to claim 19 or 22, charac- 
terized in that parameters of means for sensing said 
first viewpoint video are known, and 

the program code of said first video genera- 
tion step includes the step of generating the video 
of the virtual object viewed from said first viewpoint 
position in accordance with the known parameters. 

25. The medium according to claim 19 or 22, charac- 
terized in that some of parameters of means for 
sensing a video viewed from said first viewpoint po- 
sition are variable, 

the program code of said medium further com- 
prises the measurement step of measuring 
changes of the parameters, 
and 

the program code of said first video generation 
step includes the step of generating the video 
of the virtual object viewed from said first view- 
point position in accordance with the parame- 
ters measured in the measurement step. 



26. The medium according to claim 25, characterized 
in that the parameters of the means for sensing a 
video viewed from said first viewpoint position 
measured in the measurement step include at least 

s one of a viewpoint position/posture, and zoom ratio. 

27. The medium according to claim 19, characterized 
in that when a plurality of means for sensing a video 
viewed from said first viewpoint position are 

10 present, 

said medium further comprises a program code 
of the selection step of receiving a plurality of 
videos of the real space viewed from a first 
15 viewpoint position from the plurality of means 

for sensing a video viewed from said first view- 
point position, and outputting the video of the 
real space viewed from a first viewpoint position 
input from one selected means for sensing a 
20 video of said first viewpoint position to means 

for compositing a first viewpoint video, and 
the program code of said first video composi- 
tion step includes the step of generating a video 
of the virtual object viewed from said first view- 
25 point position using parameters of the means 

for sensing a video viewed from a first viewpoint 
position selected in the selection step, 

28. The apparatus according to claim 1 , characterized 
30 by further comprising printing means, 

in that said first video composition means out- 
puts the augmented reality video to said print- 
ing means, 

35 said printing means grabs one frame of the re- 

ceived video and prints out to the paper as a 
still image. 

29. The method according to claim 10, characterized 
40 by further comprising printing step, 

in that in said first video composition step the 
augmented reality video is output to means for 
printing, 

45 in said printing step one frame of the received 

video is grabbed and printed out to the paper 
as a still image. 

30. The method according to claim 19, characterized 
so by further comprising a program code of the printing 

step, 

in that in the program codes of said first video 
composition step the augmented reality video 
55 is output to means for printing, 

in the program codes of said printing step one 
frame of the received video is grabbed and 
printed out to the paper as a still image. 



20 



25 



30 



45 



55 



14 



7/16/2007, EAST Version: 2.1.0.14 



EP1 117 074 A2 




7/16/2007, EAST Version: 2.1.0.14 



EP1 117 074 A2 



o 




16 



7/16/2007, 



EAST Version: 2.1.0.14 



EP1 117 074 A2 






17 



7/16/2007, EAST Version: 2.1.0. 



EP1 117 074 A2 



O 




CM 




CO 


CVJ 

i 


SUBJECTIVE 
VIEWPOINT 
VIRTUAL VIDE( 


CM 


2 

o 

UJ > 0C 


IVE 

SIT VIDEO 

moN 


SUBJECT 
VIEWPOIf 
GENERA1 




SUBJECT 
VIEWPOII 
COMPOS 

UINI 1 



18 



7/16/2007, EAST Version: 2.1.0.14 



EP1 117 074 A2 



O 

1 




19 



7/16/2007, EAST Version: 2.1.0.14 




20 



7/16/2007, EAST Version: 2.1.0.14 



EP 1 117 074 A2 



FIG. 7 



701- 



702^ 



703— 



704— 



705— 




CONSOLE 



—706 



CAMERA PARAMETER 
MEASUREMENT/ 
MANAGEMENT UNIT 



GAME STATE 
MANAGER 



OBJECTIVE 
VIEWPOINT VIDEO 
GENERATOR 



-204 



-201 



—202 



OBJECTIVE VIEWPOINT 
VIDEO COMPOSITION 
UNIT 



-203 



SUBJECTIVE 
VIEWPOINT VIDEO 
GENERATOR 



—212 



SUBJECTIVE VIEWPOINT 
VIDEO COMPOSITION UNIT 



-213 



21 

7/16/2007, EAST Version: 2.1.0.14 



EP1 117 074 A2 



FIG. 8 



START' ^ 



UPDATE POSITION AND POSTURE DATA 






UPDATE INFORMATION THAT PERTAINS TO 
RENDERING OF VIRTUAL OBJECT 102 






GENERATE VIDEO OF VIRTUAL OBJECT 102 






INPUT ACTUALLY SENSED VIDEO 






COMPOSITE 
/GENERATE SUBJECTIVE AND ) 
IOBJECTIVE VIEWPOINT AR VIDEOS] 








OUTPUT SUBJECTS 
VIEWPOINT AR VID 


fE AND OBJECTIVE 
EOS 



Q END J 



22 



7/16/2007, EAST Version: 2.1 




7/16/2007, EAST Version: 2.1.0.14 



EP1 117 074 A2 



FIG. 10 



( START ) 



S1001 




S1005 
jL 



USE CAMERA 901a 




BUTTON B? 
NO 



S1003 
YES 



S1006 

1 . 



USE CAMERA 901b 




BUTTON C? 
NO 



S1004 
YES 



c 



S1007 

1 



USE CAMERA 901c 



END 



3 



24 



7/16/2007, EAST Version: 2.1.0.14 



EP1 117 074 A2 



FIG. 11 



( START ) 



S1101 




INPUT POSITION OF DISPLAY 
AREA 601 






INPUT SIZE OF DISPLAY 
AREA 601 






ASSURE DISPLAY AREA 601 
ON DISPLAY SCREEN 501 






RENDER OBJECT 
AR VIDEO ON Dl 
(GENERATE AUG 


IVE VIEWPOINT 
SPLAY AREA 601 
MENTED VIDEO) 



—S1103 



-S1104 



— S1105 



C 



END 



25 



7/16/2007, EAST Version: 2.1.0.14 



EP1 117 074 A2 



FIG. 12 

1201 1202 1203 



J , I 1 



CAMERA 


CAMERA 901a 


CAMERA 901b 


CAMERA 901c 


POSITION 


(X A ,Y A ,Z A ) 


(X B ,Y B ,Z B ) 


(Xc.Yc.Zc) 


POSTURE 


(ar A ,^A,yA) 


(<*b,/?b, 7b) 


(ax,/?c,yc) 


INTERNAL 
PARAMETER 


fA 


fB 


fc 



26 



7/16/2007, EAST Version: 2.1.0.14 



EP1 117 074 A2 




27 



7/16/2007, EAST Version: 2.1.0.14 



EP1 117 074 A2 




28 



7/16/2007, EAST Version: 2.1.0.14 



EP1 117 074 A2 




z> by LU 
</>>o 



29 



7/16/2007, EAST Version: 2.1.0.14 



EP 1 117 074 A2 



O 



CD 




30 



7/16/2007, EAST Version: 2.1.0.14 



4 



EP1 117 074 A2 




31 



7/16/2007, EAST Version: 2.1.0.14 



